{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "05_Logistic_Regression",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "metadata": {
        "id": "bOChJSNXtC9g",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Logistic Regression"
      ]
    },
    {
      "metadata": {
        "id": "OLIxEDq6VhvZ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "<img src=\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/logo.png\" width=150>\n",
        "\n",
        "In the previous lesson, we saw how linear regression works really well for predicting continuous outputs that can easily fit to a line/plane. But linear regression doesn't fare well for classification asks where we want to probabilititcally determine the outcome for a given set on inputs.\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "VoMq0eFRvugb",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Overview"
      ]
    },
    {
      "metadata": {
        "id": "qWro5T5qTJJL",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "<img src=\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/logistic.jpg\" width=270>\n",
        "\n",
        "$ \\hat{y} = \\frac{1}{1 + e^{-XW}} $ \n",
        "\n",
        "*where*:\n",
        "* $\\hat{y}$ = prediction | $\\in \\mathbb{R}^{NX1}$ ($N$ is the number of samples)\n",
        "* $X$ = inputs | $\\in \\mathbb{R}^{NXD}$ ($D$ is the number of features)\n",
        "* $W$ = weights | $\\in \\mathbb{R}^{DX1}$ \n",
        "\n",
        "This is the binomial logistic regression. The main idea is to take the outputs from the linear equation ($z=XW$) and use the sigmoid (logistic) function ($\\frac{1}{1+e^{-z}}$) to restrict the value between (0, 1). "
      ]
    },
    {
      "metadata": {
        "id": "YcFvkklZSZr9",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "When we have more than two classes, we need to use multinomial logistic regression (softmax classifier). The softmax classifier will use the linear equation ($z=XW$) and normalize it to product the probabiltiy for class y given the inputs.\n",
        "\n",
        "$ \\hat{y} = \\frac{e^{XW_y}}{\\sum e^{XW}} $ \n",
        "\n",
        "*where*:\n",
        "* $\\hat{y}$ = prediction | $\\in \\mathbb{R}^{NX1}$ ($N$ is the number of samples)\n",
        "* $X$ = inputs | $\\in \\mathbb{R}^{NXD}$ ($D$ is the number of features)\n",
        "* $W$ = weights | $\\in \\mathbb{R}^{DXC}$ ($C$ is the number of classes)\n"
      ]
    },
    {
      "metadata": {
        "id": "T4Y55tpzIjOa",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "* **Objective:**  Predict the probability of class $y$ given the inputs $X$. The softmax classifier normalizes the linear outputs to determine class probabilities. \n",
        "* **Advantages:**\n",
        "  * Can predict class probabilities given a set on inputs.\n",
        "* **Disadvantages:**\n",
        "  * Sensitive to outliers since objective is minimize cross entropy loss. (Support vector machines ([SVMs](https://towardsdatascience.com/support-vector-machine-vs-logistic-regression-94cc2975433f)) are a good alternative to counter outliers).\n",
        "* **Miscellaneous:** Softmax classifier is going to used widely in neural network architectures as the last layer since it produces class probabilities."
      ]
    },
    {
      "metadata": {
        "id": "Jq65LZJbSpzd",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Training"
      ]
    },
    {
      "metadata": {
        "id": "-HBPn8zPTQfZ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "*Steps*:\n",
        "\n",
        "1. Randomly initialize the model's weights $W$.\n",
        "2. Feed inputs $X$ into the model to receive the logits ($z=XW$). Apply the softmax operation on the logits to get the class probabilies $\\hat{y}$ in one-hot encoded form. For example, if there are three classes, the predicted class probabilities could look like [0.3, 0.3, 0.4]. \n",
        "3. Compare the predictions $\\hat{y}$ (ex.  [0.3, 0.3, 0.4]]) with the actual target values $y$ (ex. class 2 would look like [0, 0, 1]) with the objective (cost) function to determine loss $J$. A common objective function for logistics regression is cross-entropy loss. \n",
        "  * $J(\\theta) = - \\sum_i y_i ln (\\hat{y_i}) =  - \\sum_i y_i ln (\\frac{e^{X_iW_y}}{\\sum e^{X_iW}}) $\n",
        "   * $y$ = [0, 0, 1]\n",
        "  * $\\hat{y}$ = [0.3, 0.3, 0.4]]\n",
        "  * $J(\\theta) = - \\sum_i y_i ln (\\hat{y_i}) =  - \\sum_i y_i ln (\\frac{e^{X_iW_y}}{\\sum e^{X_iW}}) = - \\sum_i [0 * ln(0.3) + 0 * ln(0.3) + 1 * ln(0.4)] = -ln(0.4) $\n",
        "  * This simplifies our cross entropy objective to the following: $J(\\theta) = - ln(\\hat{y_i})$ (negative log likelihood).\n",
        "  * $J(\\theta) = - ln(\\hat{y_i}) = - ln (\\frac{e^{X_iW_y}}{\\sum_i e^{X_iW}}) $\n",
        "4. Calculate the gradient of loss $J(\\theta)$ w.r.t to the model weights. Let's assume that our classes are mutually exclusive (a set of inputs could only belong to one class).\n",
        " * $\\frac{\\partial{J}}{\\partial{W_j}} = \\frac{\\partial{J}}{\\partial{y}}\\frac{\\partial{y}}{\\partial{W_j}} = - \\frac{1}{y}\\frac{\\partial{y}}{\\partial{W_j}} = - \\frac{1}{\\frac{e^{W_yX}}{\\sum e^{XW}}}\\frac{\\sum e^{XW}e^{W_yX}0 - e^{W_yX}e^{W_jX}X}{(\\sum e^{XW})^2} = \\frac{Xe^{W_j}X}{\\sum e^{XW}} = XP$\n",
        "  * $\\frac{\\partial{J}}{\\partial{W_y}} = \\frac{\\partial{J}}{\\partial{y}}\\frac{\\partial{y}}{\\partial{W_y}} = - \\frac{1}{y}\\frac{\\partial{y}}{\\partial{W_y}} = - \\frac{1}{\\frac{e^{W_yX}}{\\sum e^{XW}}}\\frac{\\sum e^{XW}e^{W_yX}X - e^{W_yX}e^{W_yX}X}{(\\sum e^{XW})^2} = \\frac{1}{P}(XP - XP^2) = X(P-1)$\n",
        "5. Apply backpropagation to update the weights $W$ using gradient descent. The updates will penalize the probabiltiy for the incorrect classes (j) and encourage a higher probability for the correct class (y).\n",
        "  * $W_i = W_i - \\alpha\\frac{\\partial{J}}{\\partial{W_i}}$\n",
        "6. Repeat steps 2 - 4 until model performs well."
      ]
    },
    {
      "metadata": {
        "id": "r_hKrjzdtTgM",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Data"
      ]
    },
    {
      "metadata": {
        "id": "PyccHrQztVEu",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We're going to the load the titanic dataset we looked at in lesson 03_Pandas."
      ]
    },
    {
      "metadata": {
        "id": "H385V4VUtWOv",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from argparse import Namespace\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import urllib"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "pL67TlZO6Zg4",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Arguments\n",
        "args = Namespace(\n",
        "    seed=1234,\n",
        "    data_file=\"titanic.csv\",\n",
        "    train_size=0.75,\n",
        "    test_size=0.25,\n",
        "    num_epochs=100,\n",
        ")\n",
        "\n",
        "# Set seed for reproducability\n",
        "np.random.seed(args.seed)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "7sp_tSyItf1_",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Upload data from GitHub to notebook's local drive\n",
        "url = \"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/data/titanic.csv\"\n",
        "response = urllib.request.urlopen(url)\n",
        "html = response.read()\n",
        "with open(args.data_file, 'wb') as f:\n",
        "    f.write(html)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "7alqmyzXtgE8",
        "colab_type": "code",
        "outputId": "353702e3-76f7-479d-df7a-5effcc8a7461",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 272
        }
      },
      "cell_type": "code",
      "source": [
        "# Read from CSV to Pandas DataFrame\n",
        "df = pd.read_csv(args.data_file, header=0)\n",
        "df.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>pclass</th>\n",
              "      <th>name</th>\n",
              "      <th>sex</th>\n",
              "      <th>age</th>\n",
              "      <th>sibsp</th>\n",
              "      <th>parch</th>\n",
              "      <th>ticket</th>\n",
              "      <th>fare</th>\n",
              "      <th>cabin</th>\n",
              "      <th>embarked</th>\n",
              "      <th>survived</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>Allen, Miss. Elisabeth Walton</td>\n",
              "      <td>female</td>\n",
              "      <td>29.0000</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>24160</td>\n",
              "      <td>211.3375</td>\n",
              "      <td>B5</td>\n",
              "      <td>S</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>Allison, Master. Hudson Trevor</td>\n",
              "      <td>male</td>\n",
              "      <td>0.9167</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>113781</td>\n",
              "      <td>151.5500</td>\n",
              "      <td>C22 C26</td>\n",
              "      <td>S</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1</td>\n",
              "      <td>Allison, Miss. Helen Loraine</td>\n",
              "      <td>female</td>\n",
              "      <td>2.0000</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>113781</td>\n",
              "      <td>151.5500</td>\n",
              "      <td>C22 C26</td>\n",
              "      <td>S</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1</td>\n",
              "      <td>Allison, Mr. Hudson Joshua Creighton</td>\n",
              "      <td>male</td>\n",
              "      <td>30.0000</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>113781</td>\n",
              "      <td>151.5500</td>\n",
              "      <td>C22 C26</td>\n",
              "      <td>S</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1</td>\n",
              "      <td>Allison, Mrs. Hudson J C (Bessie Waldo Daniels)</td>\n",
              "      <td>female</td>\n",
              "      <td>25.0000</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>113781</td>\n",
              "      <td>151.5500</td>\n",
              "      <td>C22 C26</td>\n",
              "      <td>S</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   pclass                                             name     sex      age  \\\n",
              "0       1                    Allen, Miss. Elisabeth Walton  female  29.0000   \n",
              "1       1                   Allison, Master. Hudson Trevor    male   0.9167   \n",
              "2       1                     Allison, Miss. Helen Loraine  female   2.0000   \n",
              "3       1             Allison, Mr. Hudson Joshua Creighton    male  30.0000   \n",
              "4       1  Allison, Mrs. Hudson J C (Bessie Waldo Daniels)  female  25.0000   \n",
              "\n",
              "   sibsp  parch  ticket      fare    cabin embarked  survived  \n",
              "0      0      0   24160  211.3375       B5        S         1  \n",
              "1      1      2  113781  151.5500  C22 C26        S         1  \n",
              "2      1      2  113781  151.5500  C22 C26        S         0  \n",
              "3      1      2  113781  151.5500  C22 C26        S         0  \n",
              "4      1      2  113781  151.5500  C22 C26        S         0  "
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "metadata": {
        "id": "k-5Y4zLIoE6s",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Scikit-learn implementation"
      ]
    },
    {
      "metadata": {
        "id": "ILkbyBHQoIwE",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Note**: The `LogisticRegression` class in Scikit-learn uses coordinate descent to solve the fit. However, we are going to use Scikit-learn's `SGDClassifier` class which uses stochastic gradient descent. We want to use this optimization approach because we will be using this for the models in subsequent lessons."
      ]
    },
    {
      "metadata": {
        "id": "W1MJODStIu8V",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Import packages\n",
        "from sklearn.linear_model import SGDClassifier\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "from sklearn.model_selection import train_test_split"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "kItBIOOCTi6p",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Preprocessing\n",
        "def preprocess(df):\n",
        "  \n",
        "    # Drop rows with NaN values\n",
        "    df = df.dropna()\n",
        "\n",
        "    # Drop text based features (we'll learn how to use them in later lessons)\n",
        "    features_to_drop = [\"name\", \"cabin\", \"ticket\"]\n",
        "    df = df.drop(features_to_drop, axis=1)\n",
        "\n",
        "    # pclass, sex, and embarked are categorical features\n",
        "    categorical_features = [\"pclass\",\"embarked\",\"sex\"]\n",
        "    df = pd.get_dummies(df, columns=categorical_features)\n",
        "\n",
        "    return df"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "QwQHDh4xuYTB",
        "colab_type": "code",
        "outputId": "153ea757-b817-406d-dbde-d1fba88f194b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 224
        }
      },
      "cell_type": "code",
      "source": [
        "# Preprocess the dataset\n",
        "df = preprocess(df)\n",
        "df.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>age</th>\n",
              "      <th>sibsp</th>\n",
              "      <th>parch</th>\n",
              "      <th>fare</th>\n",
              "      <th>survived</th>\n",
              "      <th>pclass_1</th>\n",
              "      <th>pclass_2</th>\n",
              "      <th>pclass_3</th>\n",
              "      <th>embarked_C</th>\n",
              "      <th>embarked_Q</th>\n",
              "      <th>embarked_S</th>\n",
              "      <th>sex_female</th>\n",
              "      <th>sex_male</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>29.0000</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>211.3375</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0.9167</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>151.5500</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2.0000</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>151.5500</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>30.0000</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>151.5500</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>25.0000</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>151.5500</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "       age  sibsp  parch      fare  survived  pclass_1  pclass_2  pclass_3  \\\n",
              "0  29.0000      0      0  211.3375         1         1         0         0   \n",
              "1   0.9167      1      2  151.5500         1         1         0         0   \n",
              "2   2.0000      1      2  151.5500         0         1         0         0   \n",
              "3  30.0000      1      2  151.5500         0         1         0         0   \n",
              "4  25.0000      1      2  151.5500         0         1         0         0   \n",
              "\n",
              "   embarked_C  embarked_Q  embarked_S  sex_female  sex_male  \n",
              "0           0           0           1           1         0  \n",
              "1           0           0           1           0         1  \n",
              "2           0           0           1           1         0  \n",
              "3           0           0           1           0         1  \n",
              "4           0           0           1           1         0  "
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "metadata": {
        "id": "wsGRZNNiUTqj",
        "colab_type": "code",
        "outputId": "c9364be7-3cae-487f-9d96-3210b3129199",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "# Split the data\n",
        "mask = np.random.rand(len(df)) < args.train_size\n",
        "train_df = df[mask]\n",
        "test_df = df[~mask]\n",
        "print (\"Train size: {0}, test size: {1}\".format(len(train_df), len(test_df)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train size: 199, test size: 71\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "oZKxFmATU95M",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Note**: If you have preprocessing steps like standardization, etc. that are calculated, you need to separate the training and test set first before spplying those operations. This is because we cannot apply any knowledge gained from the test set accidentally during preprocessing/training."
      ]
    },
    {
      "metadata": {
        "id": "cLzL_LJd4vQ-",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Separate X and y\n",
        "X_train = train_df.drop([\"survived\"], axis=1)\n",
        "y_train = train_df[\"survived\"]\n",
        "X_test = test_df.drop([\"survived\"], axis=1)\n",
        "y_test = test_df[\"survived\"]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "AdTYbV472UNJ",
        "colab_type": "code",
        "outputId": "214a8114-3fd3-407f-cd6e-5f5d07294f50",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "cell_type": "code",
      "source": [
        "# Standardize the data (mean=0, std=1) using training data\n",
        "X_scaler = StandardScaler().fit(X_train)\n",
        "\n",
        "# Apply scaler on training and test data (don't standardize outputs for classification)\n",
        "standardized_X_train = X_scaler.transform(X_train)\n",
        "standardized_X_test = X_scaler.transform(X_test)\n",
        "\n",
        "# Check\n",
        "print (\"mean:\", np.mean(standardized_X_train, axis=0)) # mean should be ~0\n",
        "print (\"std:\", np.std(standardized_X_train, axis=0))   # std should be 1"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "mean: [-1.78528326e-17  7.14113302e-17 -5.80217058e-17 -5.35584977e-17\n",
            "  3.57056651e-17 -8.92641628e-17  3.57056651e-17 -3.79372692e-17\n",
            "  0.00000000e+00  3.79372692e-17  1.04885391e-16 -6.69481221e-17]\n",
            "std: [1. 1. 1. 1. 1. 1. 1. 1. 0. 1. 1. 1.]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "7-vm9AZm1_f9",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Initialize the model\n",
        "log_reg = SGDClassifier(loss=\"log\", penalty=\"none\", max_iter=args.num_epochs, \n",
        "                        random_state=args.seed)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "0e8U9NNluYVp",
        "colab_type": "code",
        "outputId": "c5f22ade-bb8c-479b-d300-98758a82d396",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "cell_type": "code",
      "source": [
        "# Train\n",
        "log_reg.fit(X=standardized_X_train, y=y_train)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "SGDClassifier(alpha=0.0001, average=False, class_weight=None, epsilon=0.1,\n",
              "       eta0=0.0, fit_intercept=True, l1_ratio=0.15,\n",
              "       learning_rate='optimal', loss='log', max_iter=100, n_iter=None,\n",
              "       n_jobs=1, penalty='none', power_t=0.5, random_state=1234,\n",
              "       shuffle=True, tol=None, verbose=0, warm_start=False)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "metadata": {
        "id": "hA7Oz97NAe8A",
        "colab_type": "code",
        "outputId": "ab8a878a-6012-4727-8cd1-40bc5c69245b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "cell_type": "code",
      "source": [
        "# Probabilities\n",
        "pred_test = log_reg.predict_proba(standardized_X_test)\n",
        "print (pred_test[:5])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[0.60319594 0.39680406]\n",
            " [0.00374908 0.99625092]\n",
            " [0.81886302 0.18113698]\n",
            " [0.01082253 0.98917747]\n",
            " [0.93508814 0.06491186]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "-jZtTd7F6_ps",
        "colab_type": "code",
        "outputId": "d2306e4c-88a4-4ac4-9ad5-879fa461617f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "cell_type": "code",
      "source": [
        "# Predictions (unstandardize them)\n",
        "pred_train = log_reg.predict(standardized_X_train) \n",
        "pred_test = log_reg.predict(standardized_X_test)\n",
        "print (pred_test)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[0 1 0 1 0 1 0 0 1 1 0 0 0 0 1 0 1 0 0 0 1 1 1 1 0 0 0 0 1 0 0 0 1 0 0 1 0\n",
            " 1 0 0 1 1 1 0 1 1 0 0 0 0 1 0 0 1 0 1 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "dM7iYW8ANYjy",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Evaluation metrics"
      ]
    },
    {
      "metadata": {
        "id": "uFXbczqu8Rno",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from sklearn.metrics import accuracy_score"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "sEjansj78Rqe",
        "colab_type": "code",
        "outputId": "f5bfbe87-12c9-4aa5-fc61-e615ad4e63d4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "# Accuracy\n",
        "train_acc = accuracy_score(y_train, pred_train)\n",
        "test_acc = accuracy_score(y_test, pred_test)\n",
        "print (\"train acc: {0:.2f}, test acc: {1:.2f}\".format(train_acc, test_acc))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "train acc: 0.77, test acc: 0.82\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "WijzY-vDNbE9",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "So far we looked at accuracy as the metric that determines our mode's level of performance. But we have several other options when it comes to  evaluation metrics.\n",
        "\n",
        "<img src=\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/metrics.jpg\" width=400>"
      ]
    },
    {
      "metadata": {
        "id": "80MwyE0yOr-k",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The metric we choose really depends on the situation.\n",
        "positive - true, 1, tumor, issue, etc., negative - false, 0, not tumor, not issue, etc.\n",
        "\n",
        "$\\text{accuracy} = \\frac{TP+TN}{TP+TN+FP+FN}$ \n",
        "\n",
        "$\\text{recall} = \\frac{TP}{TP+FN}$ → (how many of the actual issues did I actually catch)\n",
        "\n",
        "$\\text{precision} = \\frac{TP}{TP+FP}$ → (out of all the things I said were issues, how many were actually issues)\n",
        "\n",
        "$F_1 = 2 * \\frac{\\text{precision }  *  \\text{ recall}}{\\text{precision } + \\text{ recall}}$\n",
        "\n",
        "where: \n",
        "* TP: # of samples predicted to be positive and were actually positive\n",
        "* TN: # of samples predicted to be negative and were actually negative\n",
        "* FP: # of samples predicted to be positive but were actually negative\n",
        "* FN: # of samples predicted to be negative but were actually positive"
      ]
    },
    {
      "metadata": {
        "id": "opmu3hJm9LXA",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import itertools\n",
        "from sklearn.metrics import classification_report, confusion_matrix"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "wAzOL8h29m82",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Plot confusion matrix\n",
        "def plot_confusion_matrix(cm, classes):\n",
        "    cmap=plt.cm.Blues\n",
        "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
        "    plt.title(\"Confusion Matrix\")\n",
        "    plt.colorbar()\n",
        "    tick_marks = np.arange(len(classes))\n",
        "    plt.xticks(tick_marks, classes, rotation=45)\n",
        "    plt.yticks(tick_marks, classes)\n",
        "    plt.grid(False)\n",
        "\n",
        "    fmt = 'd'\n",
        "    thresh = cm.max() / 2.\n",
        "    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
        "        plt.text(j, i, format(cm[i, j], 'd'),\n",
        "                 horizontalalignment=\"center\",\n",
        "                 color=\"white\" if cm[i, j] > thresh else \"black\")\n",
        "\n",
        "    plt.ylabel('True label')\n",
        "    plt.xlabel('Predicted label')\n",
        "    plt.tight_layout()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "KqUVzahQ-5ic",
        "colab_type": "code",
        "outputId": "bff8819e-3d5b-45b9-c221-179c873140b1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 520
        }
      },
      "cell_type": "code",
      "source": [
        "# Confusion matrix\n",
        "cm = confusion_matrix(y_test, pred_test)\n",
        "plot_confusion_matrix(cm=cm, classes=[\"died\", \"survived\"])\n",
        "print (classification_report(y_test, pred_test))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "             precision    recall  f1-score   support\n",
            "\n",
            "          0       0.74      0.91      0.82        32\n",
            "          1       0.91      0.74      0.82        39\n",
            "\n",
            "avg / total       0.83      0.82      0.82        71\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbIAAAGACAYAAADIwee6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3XlclPX6//H3sAlkKuCKay4nLcU8\nZYUeN9RIk7KsRA09Vl9Xcilz17TMrdSjaW7HXPBUEmpiuGt61EBz6aDmlnbMXEgRVBDZf3/4iyOp\n97gNc9/0evqYx4O5Z+77vij0zfW5r5mx5ebm5goAAItycXYBAADcC4IMAGBpBBkAwNIIMgCApRFk\nAABLI8gAAJZGkMFScnNzNX/+fLVp00bBwcFq0aKFRo0apcuXL9/TcQcMGKAmTZpo69atd7xvfHy8\n3njjjXs6//UGDx6s2rVrKzk5Od/2Xbt26eGHH9ayZcvsHmPVqlVKSUm56WOTJk3SF198cV9qBcyA\nIIOlfPzxx1q1apXmzZuntWvXKjo6WpmZmerevbvu5SWRMTExioiIUKNGje5434CAAM2bN++uz30z\nJUuW1Nq1a/Nti4mJUbly5W5r/2nTpt0yyN555x116NDhnmsEzIIgg2UkJycrIiJC48ePV5kyZSRJ\n3t7eGjlypN58803l5uYqPT1dI0eOVHBwsFq1aqXx48crOztbkhQUFKQvv/xSL7/8sv72t79p/Pjx\nkqSwsDDl5OTojTfe0JYtWxQUFKRdu3blnff3+1lZWRo2bJiCg4PVsmVLhYeHKyUlRTt27FDLli0l\n6a7OfzONGzfWN998k3c/OztbW7du1V//+te8bcePH1eHDh3UqlUrtWzZMu/5Q4YM0c8//6ywsDDt\n2rVLgwcP1rhx4xQSEqLVq1dr8ODB+vTTTxUfH6+mTZsqNTVVkjRr1iz16dPnnv8/AQWNIINl/Oc/\n/1HZsmVVrVq1fNuLFCmioKAgubi4aOHChTp79qxiYmK0fPly7dq1K18gfP/991qyZImWLl2qxYsX\n6+zZs4qIiJAkRUREqEmTJrc8/7Zt2/Trr79qzZo1WrdunapXr669e/fme87dnP9m6tatq1OnTikh\nIUGSFBsbq4CAAHl4eOQ9Z+LEiWrWrJlWr16tsWPHatiwYcrMzNS4cePyvp8nnngib/+oqCi1atUq\nb/+AgAC1aNFCs2fPVkJCgj7//HMNHz781v8DAJMiyGAZycnJ8vPzM3zO5s2b9eqrr8rNzU2enp4K\nCQnR9u3b8x4PCQmRq6urypQpIz8/P505c+a2z+/r66tjx45p/fr1SktLU79+/W5Yirxf57fZbAoO\nDlZMTIyka8uKrVu3zvecTz/9NO/a3OOPP6709HSdO3fupscLDAxUkSJFbtjev39/rVmzRkOGDFGv\nXr1UunTp2/7vAZgFQQbL8PHxyetQbuXChQsqXrx43v3ixYsrMTEx737RokXzvnZ1dc1b9rsdAQEB\nGj58uCIiItSwYUO98847unTpksPO36ZNG33zzTfKyMjQjh071Lhx43yPb926VZ06dVJwcLBat26t\n3Nxc5eTk3PRY19d0vQceeECtWrXS7t27FRIScutvHjAxggyW8dhjjykxMVEHDhzItz0zM1NTpkxR\nWlqaSpYsmW/aLzk5WSVLlryj87i4uOQLhIsXL+Z9/eyzzyoiIkLffvut0tLSbhjyuB/n/92jjz6q\n1NRURUZGqn79+vmWFTMzM9WvXz/17Nkzb+jFZrPd8TkSEhK0cuVKPffcc5o+ffpd1Qk4G0EGyyhW\nrJjefPNNDRo0SCdOnJAkpaWlaeTIkfrxxx/l5eWlpk2bKioqStnZ2bpy5YpWrFhheN3rZkqVKqVD\nhw5JujbGnp6eLklaunSpZsyYIUkqUaKEqlatesO+9+P813vuuec0c+bMG5YV09LSdOXKFdWuXVvS\ntWtz7u7uunLliiTJzc3thm7xZj788EO9+eabGjp0qFavXq2DBw/eda2AsxBksJS33npLr776qnr2\n7Kng4GC99NJL8vPzy+smwsLCVLZsWT333HNq166dmjZtmm/A4Xb06tVLCxYsUJs2bXTs2DFVr15d\nktS8eXMdOHBAzzzzjFq1aqWffvpJXbt2zbfv/Tj/9Z577jllZWWpQYMG+bb/Hupt27ZV27ZtValS\nJbVo0UI9evTQlStX9Oyzzyo0NFSrVq265bE3b96sX3/9VaGhoSpatKj69++v4cOH39FyK2AGNj6P\nDABgZXRkAABLI8gAAJZGkAEALI0gAwBYGkEGALA0N2cXYAZe9cKdXQIKoaTveYExHMOzgP/lvpd/\nI9P2Ov7vAUEGADBmM/finbmrAwDADjoyAICxu3gfz4JEkAEAjJl8aZEgAwAYoyMDAFgaHRkAwNLo\nyAAAlmbyjszc1QEAYAcdGQDAGEuLAABLM/nSIkEGADBGRwYAsDQ6MgCApdGRAQAszeQdmbmrAwDA\nDjoyAIAxk3dkBBkAwJgL18gAAFZGRwYAsDSmFgEAlmbyjszc1QEAYAcdGQDAGEuLAABLM/nSIkEG\nADBGRwYAsDQ6MgCApdGRAQAszeQdmbmrAwDADjoyAIAxlhYBAJZm8qVFggwAYIwgAwBYGkuLAABL\noyMDAFiayTsyc8csAAB20JEBAIyxtAgAsDSTLy0SZAAAQzaCDABgZQQZAMDazJ1jBBkAwHkmTpyo\n3bt3KysrS927d9emTZt04MABlShRQpL0xhtvqGnTpobHIMgAAIYctbQYFxeno0ePasmSJUpKStKL\nL76op59+Wm+//baaNWt228chyAAAhhwVZPXr11dAQIAkqVixYkpLS1N2dvYdH8fcLw4AADidzWa7\n65sRV1dXeXt7S5KioqLUuHFjubq6avHixercubP69++vCxcu2K2PjgwAYMjRU4sbNmxQVFSUPvvs\nM+3fv18lSpRQrVq1NGfOHE2fPl0jR4403J+ODABgzHYPNzu2bt2qWbNmae7cuXrwwQcVGBioWrVq\nSZKCgoJ05MgRu8cgyAAAhhy1tHj58mVNnDhRs2fPzptSfOutt3Ty5ElJ0o4dO1SjRg279bG0CABw\nilWrVikpKUn9+vXL2/bSSy+pX79+8vLykre3t8aNG2f3OLbc3NxcRxZqBV71wp1dAgqhpO+nO7sE\nFFKeBdyC+Lz2r7veN2lxp/tYyc3RkQEADPEWVQAASyPIAADWZu4cI8gAAMboyAAAlmb2ION1ZAAA\nS6MjAwAYMntHRpABAIyZO8cIMgCAMToyAIClEWQAAEsjyAAAlmb2IGP8HgBgaXRkAABj5m7ICDIA\ngDGWFmFZzzWpo7gvB2vv0uHa+Fl/PVKtnFxdXTT+7Rf1w7LhOrzqffXv3NzZZcLili9bqqcef0x1\na9dUUJO/6cD+/c4uCX/gqE+Ivl/oyHBT/qWKa+77YQrqOlmHjp9Vt1caafrwUH2x6nvVr11FT4WO\nVxEPN21ZOEA79/1X2/cec3bJsKBffvlFfXr30La4XapcubKmT5uq7v/3urbF7nR2abgOHRksKTMr\nW12GzNeh42clSd/9cEy1qpVT86dqasnqXUrPyNKllKuKiI5T2+aPOblaWJW7u7sWRHyuypUrS5Ka\nBTXX0SOHnVwVbmC7h1sBoCPDTZ1LStH67w7m3Q9u+Ii+33dCuZJcXf/3+0/KlXRVrVjKCRWiMChX\nrpzKlSsnScrKylLEogVqE/KCk6uC1dCRwa6mT/5F4Z2CNHDSUm2MO6QubQNVvKiXfIs/oI5tnpRn\nEX4fwr2ZPm2qKpcvo+3btmrMuAnOLgd/YPZrZJYKstTUVAUFBal///66evXqbe3Tp08f7dixw8GV\nFV4hTQM0d3SY2vWdpUPHz2r+8u+0Ke6Q/h0xQF98/KY2xh3Sxctpzi4TFhfep69+PXte4X36qVnj\nBkpL42fKTAgyB5gyZYo8PT2dXUah1+yph/XxwJfVptd07fnxF0lSdnaOhv7ja9V98QMF/99UZWVn\na//R006uFFZ16OBBbdq4QdK1fyzbh3bQ5UuXdOQw18nMxOxBZvo1oZSUFL311ltKT0/X448/LkkK\nCgrSypUrlZKSomHDhikzM1Ourq4aM2aM/P39NXfuXMXExMjf318pKSlO/g6sycvTXXNGvaZX356j\nwz8n5G0PbfWEWjepoy5DFqhsyWIKC3labXpNd2KlsLLz58/pja6dtT1ul/z9/fXd9u3KzMzUQ1Wr\nOrs0XMfsU4umD7IVK1aoRo0aGjp0qFatWqWYmJi8x6ZOnarXX39dDRo00JYtW/Tpp59q4MCB+uKL\nL7R69WplZmaqZcuWTqzeukKaBqikT1HN/7BLvu3P956hts0f048r31NWdo5GTFuh4yfPO6lKWN3f\nGjXWoMHD9NyzLZSTk6MiHkW06F9fqlixYs4uDdczd46ZP8iOHTum+vXrS5KefPLJfI/t3btXP//8\ns2bOnKns7Gz5+vrqxIkTql69uooUKaIiRYro0UcfdUbZlhe5Zrci1+y+6WOhA/5ZwNWgMOvRq7d6\n9Ort7DJggI7sHuXm5srF5dqlvJycnHyPubu7a+rUqSpdunTetvj4+Lzn/74/AKDwMv2wx0MPPaT9\n//8ta/44fVi3bl1t2HDtQnFsbKxWrlypSpUq6dixY8rIyFBKSkrevgCAu8Owxz1q27atevfurS5d\nuuQNe/wuPDxcQ4cOVUxMjGw2m8aNG6cSJUqobdu2Cg0NVYUKFVSnTh0nVQ4AhYPJVxZly2XtTV71\nwp1dAgqhpO+Z5oRjeBZwC1Lj3TV3ve/Rj569j5XcnOk7MgCAc5m9IyPIAACGmFoEAFiayXPM/FOL\nAAAYoSMDABhycTF3S0aQAQAMmX1pkSADABhi2AMAYGkmzzGCDABgzOwdGVOLAABLoyMDABgye0dG\nkAEADJk8xwgyAIAxOjIAgKWZPMcIMgCAMToyAIClmTzHGL8HAFgbHRkAwBBLiwAASzN5jhFkAABj\ndGQAAEtzZI5NnDhRu3fvVlZWlrp37646depo4MCBys7OVqlSpfTRRx/Jw8PD8BgEGQDAkKM6sri4\nOB09elRLlixRUlKSXnzxRQUGBqpjx45q1aqVJk+erKioKHXs2NHwOEwtAgAM2Wx3fzNSv359TZ06\nVZJUrFgxpaWlaceOHWrevLkkqVmzZoqNjbVbH0EGAHAKV1dXeXt7S5KioqLUuHFjpaWl5S0l+vn5\n6dy5c3aPQ5ABAAzZbLa7vt2ODRs2KCoqSiNHjsy3PTc397b25xoZAMCQI4c9tm7dqlmzZumf//yn\nHnzwQXl7e+vq1avy9PRUQkKCSpcubfcYdGQAAEOO6sguX76siRMnavbs2SpRooQkqUGDBlq7dq0k\nad26dWrUqJHd+ujIAACGHDW1uGrVKiUlJalfv35528aPH6/hw4dryZIl8vf3V9u2be0ehyADABhy\n1NJi+/bt1b59+xu2z58//46OQ5ABAAyZ/Z09uEYGALA0OjIAgCGTN2QEGQDAmNmXFgkyAIAhk+cY\nQQYAMOZi8iQjyAAAhkyeY0wtAgCsjY4MAGCIYQ8AgKW5mDvHCDIAgDE6MgCApZk8xwgyAIAxm8yd\nZAQZAMCQ2a+RMX4PALA0OjIAgCGGPQAAlmbyHCPIAADGLPtei1FRUYY7vvzyy/e9GACA+Zg8x24d\nZLt37zbckSADgD8Hy14jGzduXN7XOTk5SkxMVKlSpQqkKACAeZg8x+yP38fGxqpFixYKCwuTJI0d\nO1abN292dF0AANwWu0E2ZcoURUZG5nVjPXr00KeffurwwgAA5uBis931rSDYnVr09vZWyZIl8+77\n+vrK3d3doUUBAMzD5CuL9oPM09NTO3fulCRdvHhRMTExKlKkiMMLAwCYg9mHPewuLb733nuaN2+e\n9u3bp5YtW2rr1q16//33C6I2AIAJuNju/lYQ7HZk5cqV0+zZswuiFgCACVm+I/v+++/Vrl07PfbY\nY6pXr57at29v9zVmAIDCw2a7+1tBsNuRvf/++xo6dKj++te/Kjc3V7t379bo0aMVHR1dEPUBAGDI\nbpD5+fkpMDAw737Dhg3l7+/v0KIAAOZh9qXFWwbZyZMnJUl16tTRZ599pgYNGsjFxUWxsbF65JFH\nCqxAAIBzmf2DNW8ZZF26dJHNZlNubq4kafHixXmP2Ww29enTx/HVAQCczrId2aZNm2650549exxS\nDADAfMwdY7dxjSwlJUUrVqxQUlKSJCkzM1NLly7Vtm3bHF4cAMD5zP55ZHbH7/v166fDhw9r2bJl\nSk1N1bfffqtRo0YVQGkAANhnN8jS09P1/vvvq3z58ho0aJAWLVqk1atXF0RtAAATsPzryDIzM3Xl\nyhXl5OQoKSlJPj4+eRONAIDCz7LDHr974YUXFBkZqVdeeUWtW7eWr6+vKlWqVBC1AQBMwOQ5Zj/I\nOnTokPd1YGCgEhMTeR0ZAPyJmH3Y45ZBNnXq1FvutH79evXt29chBQEAzMXkOXbrIHN1dS3IOpxq\nzZd8LA3uP5/64c4uAYVU2t7pBXo+y14jCw/nLyEAwPzsXiMDAPy52X2dlpMRZAAAQ2ZfWrytoE1K\nStK+ffskSTk5OQ4tCABgLi62u78VSH32nvDNN9+offv2GjJkiCTpgw8+0FdffeXwwgAA5mD5IJs/\nf75WrFghHx8fSdKgQYMUGRnp8MIAAOZgs9nu+lYQ7F4je/DBB+Xl5ZV339PTU+7u7g4tCgBgHpb9\nYM3f+fj4aPny5UpPT9eBAwe0atUq+fr6FkRtAADYZXdpcfTo0dq3b59SU1M1fPhwpaena8yYMQVR\nGwDABBz57vdHjhxRixYttHjxYknS4MGDFRISorCwMIWFhWnz5s12j2G3IytWrJhGjhxpvxoAQKHk\nqPdavHLlij744AMFBgbm2/7222+rWbNmt30cu0HWpEmTm16wu52UBABYn6NeEO3h4aG5c+dq7ty5\n93Qcu0H2+eef532dmZmp2NhYpaen39NJAQDW4ajhQzc3N7m53RhDixcv1vz58+Xn56cRI0bYncuw\nG7Tly5fPu1WpUkUdOnTQ1q1b775yAICluNhsd327Uy+88IIGDBigRYsWqVatWpo+3f4bJNvtyGJj\nY/PdP3v2rH755Zc7Lg4AAHuuv14WFBSkUaNG2d3HbpB9+umneV/bbDYVLVpUo0ePvrsKAQCWU5Bv\ntfjWW29p4MCBqlixonbs2KEaNWrY3cdukA0ePFiPPvrofSkQAGA9jnpB9P79+zVhwgSdOnVKbm5u\nWrt2rV577TX169dPXl5e8vb21rhx4+wex26QTZgwQYsWLbovRQMArMdR4/e1a9dWRETEDduDg4Pv\n6Dh2g8zf319hYWGqW7duvrem6tu37x2dCABgTSb/FBf7QVahQgVVqFChIGoBAJiQZd9rMTo6Ws8/\n/7zCw8MLsh4AgMnYZO4ku+XryKKiogqyDgAA7ordpUUAwJ+bZZcW9+7dq6ZNm96wPTc3Vzabjfda\nBIA/CcsG2SOPPKLJkycXZC0AABMqqE96vlu3DDIPDw+VL1++IGsBAJiQZTuygICAgqwDAGBSJm/I\nbh1k7777bkHWAQAwKUe9s8f94qjPSwMAoEAwfg8AMGTZa2QAAEgWvkYGAIAkuZj8LaoIMgCAIToy\nAIClcY0MAGBpjN8DAOBAdGQAAEMmb8gIMgCAMbMvLRJkAABDJs8xggwAYMzswxQEGQDAkNk/j8zs\nQQsAgCE6MgCAIXP3YwQZAMAOphYBAJZm7hgjyAAAdpi8ISPIAADGzD61SJABAAyZfbzd7PUBAGCI\njgwAYIilRQCApZk7xggyAIAddGQAAEsz+zAFQQYAMERHBgCwNHPHmPk7RgAADNGRAQAMmXxlkSAD\nABhzMfniIkEGADBERwYAsDQbHRkAwMrM3pExtQgAsDQ6MgCAIYY9AACWZvalRYIMAGCIIAMAWBpT\niwAAS3Mxd44xtQgAMGa7hz/2HDlyRC1atNDixYslSWfOnFFYWJg6duyovn37KiMjw+4xCDIYysrM\n1KfjR6hpTT/9dvZU3vavFs5S59ZPK+zZJzVxeF9l3sYPG/C755rUUdyXg7V36XBt/Ky/HqlWTq6u\nLhr/9ov6YdlwHV71vvp3bu7sMuFgV65c0QcffKDAwMC8bdOmTVPHjh31+eefq3LlyoqKirJ7HIIM\nhob1fk1e3g/k23bgh++1dNFszfhyrRat3qGUSxe1NGKOkyqE1fiXKq6574fp70MXqF67MVqyepem\nDw/V6y81UP3aVfRU6HjVf3WsOr8QqIb1qjm7XOjasMfd3ox4eHho7ty5Kl26dN62HTt2qHnza7/E\nNGvWTLGxsXbrI8hgqHPPAeraZ3C+bVvWRKtZ67Z6sFhx2Ww2tW7XSVvWrnBShbCazKxsdRkyX4eO\nn5UkfffDMdWqVk7Nn6qpJat3KT0jS5dSrioiOk5tmz/m5GohOW5p0c3NTZ6envm2paWlycPDQ5Lk\n5+enc+fO2a2PIIOhR+vVv2Hbyf8eU/mKD+Xd969YRb8cP1qQZcHCziWlaP13B/PuBzd8RN/vO6Fc\nSa6u//snKeVKuqpWLOWECvFHLra7v92L3Nzc26vv3k6DP6P0q1fkUaRI3v0inl5KS7vixIpgVU2f\n/IvCOwVp4KSl2hh3SF3aBqp4US/5Fn9AHds8Kc8iDFabgSOHPf7I29tbV69elSQlJCTkW3a8FVMH\n2Zw5c7R379673n/x4sX65JNP7mNFkCRPrweUkZ6ed//q1Ss3XEcD7AlpGqC5o8PUru8sHTp+VvOX\nf6dNcYf074gB+uLjN7Ux7pAuXk5zdpmQ466R3UyDBg20du1aSdK6devUqFEju/uY+tedbt26ObsE\n3ESlqjV06pfjefdP/fe4qlR72IkVwWqaPfWwPh74str0mq7DPydIkrKzczT0H19r6D++liQN6fas\n9h897cwy8f856mVk+/fv14QJE3Tq1Cm5ublp7dq1+vjjjzV48GAtWbJE/v7+atu2rd3jOCTITp8+\nrXfffVcuLi7Kzs5WgwYNlJqaqkGDBik1NVUhISHatGmTnnnmGTVu3Fh+fn76+uuv81J4+fLlOnTo\nkC5evKjg4GBNmzZNM2bMkL+/v06dOqW33npLX331lUaMGKGTJ08qKytLffr0UWBgoGJjYzV27FiV\nLFlSpUqVUsWKFR3xLf6pNX32BY3s00Wv/L2XipXwVVTEbDV/7iVnlwWL8PJ015xRr+nVt+fkhZgk\nhbZ6Qq2b1FGXIQtUtmQxhYU8rTa9pjuxUjha7dq1FRERccP2+fPn39FxHBJka9euVYMGDdS7d28d\nOHBA27dvV2pq6g3Py8rKUuPGjdW4cWPFxcXp6NGjqlGjhjZu3KjXX39dkZGRkqQWLVro22+/VadO\nnbRx40Y988wzWrlypUqVKqWxY8fqwoUL6tKli1auXKlJkybpo48+Us2aNfV///d/BNk9uHD+N/UN\nC8m736/zC3J1ddXkBcvVvmtv9enURrnK1RMNmuj5Dq87sVJYSUjTAJX0Kar5H3bJt/353jPUtvlj\n+nHle8rKztGIaSt0/OR5J1WJ67mY/M0WHRJkDRs2VHh4uC5fvqzg4GCVLFlSSUlJN31uQECAJOmZ\nZ57Rt99+q0qVKuno0aOqV69eXpA988wzGj9+fF6QjRo1SgsWLNDu3bu1Z88eSVJ6eroyMjJ06tQp\n1axZU5JUv359pV93LQd3xrdkaUWs3nHTx9p17q52nbsXcEUoDCLX7Fbkmt03fSx0wD8LuBrcDnPH\nmIOC7C9/+YtWrFih7du3a/LkyXrppf8tO2VlZeV7rru7u6RrXVe/fv1Uo0YNNWrUSLbrfgOoUaOG\nfvvtN505c0aXL1/WQw89JHd3d/Xo0UNt2rTJdzwXl//Nr9zu6CYAwIDJk8whU4sxMTE6evSoWrRo\nob59++qzzz7Tb7/9Jknavfvmv4mVKVNGNptN33zzjYKDg294vGnTppoyZYqCgoIkSXXr1tXGjRsl\nSYmJiZo8eXLecY4fP67c3Fzt3LnTEd8eAPypFOT4/d1wSEdWpUoVvffee/L29parq6s++ugjDR06\nVGFhYWrSpEm+but6QUFBWrRokT766KMbHmvZsqVCQ0MVHR0tSWrVqpXi4uIUGhqq7OxshYeHS5L6\n9eunvn37yt/fX2XLlnXEtwcAfyomv0QmWy7rb9py+IKzS0Ah9GzoSGeXgEIqbW/BTnN+f/ziXe9b\nv2rx+1jJzZn6BdEAANhj6hdEAwBMwORLiwQZAMBQQQ1t3C2CDABgyOzDHgQZAMCQyXOMIAMA2GHy\nJGNqEQBgaXRkAABDDHsAACyNYQ8AgKWZPMcIMgCAHSZPMoIMAGCIa2QAAEsz+zUyxu8BAJZGRwYA\nMGTyhowgAwDYYfIkI8gAAIYY9gAAWJrZhz0IMgCAIZPnGEEGALDD5EnG+D0AwNLoyAAAhhj2AABY\nGsMeAABLM3mOEWQAADtMnmQEGQDAkNmvkTG1CACwNDoyAIAhhj0AAJZm8hwjyAAAdpg8yQgyAIAh\nsw97EGQAAENcIwMAWJrJc4zxewCAtdGRAQCMmbwlI8gAAIYY9gAAWBrDHgAASzN5jhFkAABjdGQA\nAIszd5Ixfg8AsDQ6MgCAIZYWAQCWZvIcI8gAAMYc1ZHt2LFDffv2VY0aNSRJf/nLXzRixIg7Pg5B\nBgAw5MgXRD/55JOaNm3aPR2DIAMAGDP52iJTiwAAQ7Z7uNnz008/qUePHurQoYO2b99+V/XRkQEA\nnKJKlSoKDw9Xq1atdPLkSXXu3Fnr1q2Th4fHHR2HjgwAYMhmu/ubkTJlyqh169ay2WyqVKmSSpYs\nqYSEhDuujyADABiy3cMfI9HR0Zo3b54k6dy5c0pMTFSZMmXuuD6WFgEAxhw07BEUFKQBAwZo48aN\nyszM1KhRo+54WVEiyAAAdjhqaLFo0aKaNWvWPR+HIAMAGDL7W1RxjQwAYGl0ZAAAQ458Z4/7gSAD\nABhiaREAAAeiIwMAGDJ7R0aQAQAMcY0MAGBpZu/IuEYGALA0OjIAgCGTN2QEGQDADpMnGUEGADDE\nsAcAwNLMPuxBkAEADJk8xwhcnjKjAAAMjklEQVQyAIAdJk8yxu8BAJZGRwYAMGT2YQ9bbm5urrOL\nAADgbrG0CACwNIIMAGBpBBkAwNIIMgCApRFkAABLI8gAAJZGkAEALI0gAwBYGkGGO3b8+HH99NNP\n4rX0KCj//e9/FR8f7+wyYFK8RRXuSHZ2tv71r38pMzNTnTt3VrVq1WQz+2c8wNJSU1O1Zs0aJSYm\nysXFRbVr13Z2STAZOjLctpycHLm6umrEiBFKT09XZGSkTpw44eyyUIjl5ubqgQce0AsvvKCiRYsq\nJiZGhw8fdnZZMBk6Mtw2F5drv/csW7ZMV69e1cGDB/Xbb78pPDxc1atXd3J1KIx+7/a/++47/fzz\nzzp//rxycnLUpk0b1alTx8nVwSzoyHBHTp48qWXLlmnq1KmKjo6Wj4+PIiIi6MzgMEeOHNGiRYs0\nZswYDRw4UL6+vlqzZo2OHDni7NJgEgQZDP1xoMPb21vJycnavHmzJGnQoEE6fvy4xo0bp2PHjjmh\nQhQ2f/yZ8/LyUpkyZVS0aFEFBAQoMDBQx44d08KFC3Xo0CEnVQkzIchwS7m5uXlLO+vXr9f27duV\nkJCg4cOHa/ny5dq2bZs8PT3Vrl07ubu7y9fX18kVw+qu/5nbs2eP9uzZI39/f5UvX15hYWGSpICA\nANWuXVsPPvigSpcu7cxyYRJ8Hhlu6fd/VCIjI7V27VqFhIRo4cKFevvtt5WcnKyZM2fq6aef1uHD\nhzV+/HhVrFjR2SWjkFiyZIlWr16t2rVr69///reio6PVp08fXbp0SQ0bNtS///1vTZw4UeXKlXN2\nqTABOjLcYN++fbpw4YJycnJ04cIFbd68WVOmTFFycrJq1qypRo0aqWHDhpo6daoefvhhjRs3jhDD\nPbn+9+mTJ09q3bp1mjdvnqpXr64yZcpIkqZNm6YXX3xR7u7uGjVqFCGGPHRkyOfgwYPq2rWrtm3b\nppycHHl4eGjcuHHKzs5WcnKyxo4dq5SUFK1bt06hoaHOLheFzJYtW/T4449r8eLFOn78uNLT0/WP\nf/xDhw8f1u7du9WpUydnlwgToiNDPuXLl5e/v7/69Omjdu3a6dy5c6pYsaLWrl2rd955Rx4eHlq/\nfr2+++47ZWRkOLtcFDIxMTFasGCBMjMzdfbsWXXr1k02m00HDx7UmTNnlJ2d7ewSYUJ0ZJD0v6Ud\nm82m7du3a/r06dq7d6/+9re/adKkSYqIiNCPP/6oYsWK6dSpUxo1apSqVavm5KphZVeuXJG3t3e+\nbZs3b9bRo0f1/PPPa8GCBbp8+bKuXr2qU6dO6f3331eNGjWcVC3MjCBDPp9//rn+85//qG7dutq4\ncaO2b9+uhg0batKkSUpOTtb58+dVtmxZVahQwdmlwsKOHTum+Ph4vfjii4qOjlZOTo4CAgJUoUIF\n9erVS23atFFISIiOHj2qQ4cO6YknnuBnDrdEkEHStY7s6tWrCg8PV5cuXdS4cWMlJSVp2LBh2rRp\nk2rWrKkZM2aofPnyzi4VhcDFixdls9mUkZGhr7/+WmfPnlVWVpYqVaqkli1basaMGXr33Xfl5+fn\n7FJhAVwjg6RrS4peXl4KDAzU8ePH9dtvv8nHx0cdO3ZU9erVdebMGd7tHvdFbm6uihcvrgMHDmjk\nyJFyc3PT888/r549e2r37t2aM2eOdu7cqYMHDzq7VFgEQYZ8goODdezYMcXFxSklJUXZ2dkKCQnR\nsmXLWNrBfWGz2XT+/Hnt3LlTbdu21enTp7VhwwZJ0owZM/Tyyy+rWbNmKlu2rJMrhVWwtIgbHDt2\nTF9++aXOnDmj06dPa9q0aYQY7sn179ghSVlZWZo0aZKKFi2qnj17auzYsSpatKhCQkIYIsIdI8hw\nU6mpqUpISJCnp6f8/f2dXQ4s7PoQ+/rrr+Xm5qY2bdooIyNDPXv21N///nfVqVNH48aNU9WqVdW1\na1d5eHg4uWpYCUEGoEDEx8fr66+/1oEDB9SoUSNVqVJFxYoVU0JCgl555RUlJCTIxcVFpUqVcnap\nsBg+jwyAQ1zfiS1fvlwbNmzQxx9/rIyMDO3du1cbN27Ujz/+qJycHNWuXVu1atVycsWwKoY9ADjE\n7yF24cIFlStXTrt371ZcXJyKFy+upk2b6oMPPtBLL72khx56SJ6enk6uFlbG0iIAh8jJydGpU6fU\nrl07LViwQImJiRo9erQmT56sgICAvOdlZGRwTQz3hI4MgENkZWWpYsWKGjVqlPr166dSpUppxIgR\nGjhwoPbs2ZP3PEIM94ogA3BfXL+488MPP2jGjBm6fPmyWrdurQEDBqhXr14qX768+vfvrw8++EDp\n6elOrBaFCUuLAO7Z9YMdK1as0IEDB5Seni5/f3+1b99eJUqU0Jw5czR79mwtXbpUpUqV0gMPPODk\nqlFY0JEBuGe/h1hkZKRiYmIUGBioRo0ayWazafHixZKkOnXq6NVXX5XNZiPEcF8xfg/gvrh48aJW\nr16td955RyVLltT27dvl6empTZs2ae/evUpJSdEnn3yi0qVLO7tUFDIsLQK4L7KzszVz5kzt3LlT\n3t7eqlmzpipXrqz4+HjVr19fNWvWVNWqVZ1dJgohggzAfXP58mX98ssvqlq1qry8vLRlyxZFRkZq\n2rRpcnV1dXZ5KKQIMgD33Z49e7Rlyxbt2LFDH374IW8EDIciyADcd0lJSYqPj1eVKlVUuXJlZ5eD\nQo4gAwBYGuP3AABLI8gAAJZGkAEALI0gAwBYGkGGQuPXX39V7dq1FRYWprCwMIWGhuqdd97RpUuX\n7vqYX331lQYPHixJ6t+/vxISEm753D179ujkyZO3feysrCw9/PDDN2z/5JNPNGXKFMN9g4KCdOLE\nids+1+DBg/XVV1/d9vMBKyHIUKj4+voqIiJCERER+vLLL1W6dGnNnDnzvhx7ypQpKlOmzC0fX7Zs\n2R0FGYD7g/daRKFWv359LVmyRNK1LqZVq1Y6efKkpk2bplWrVmnx4sXKzc2Vr6+vxowZIx8fH/3r\nX//SF198obJly+Z7X8CgoCDNnz9fFStW1JgxY7R//35JUteuXeXm5qY1a9YoPj5eQ4YMUeXKlTV6\n9GilpaXpypUrevvtt9WgQQMdP35c7777rry8vPTUU0/Zrf/zzz/XihUr5O7uriJFimjKlCkqVqyY\npGvd4r59+5SYmKgRI0boqaee0unTp296XqAwI8hQaGVnZ2v9+vV6/PHH87ZVqVJF7777rs6cOaNZ\ns2YpKipKHh4eWrhwoWbPnq3evXtr2rRpWrNmjXx8fNSzZ08VL14833Gjo6N1/vx5RUZG6tKlSxow\nYIBmzpypWrVqqWfPngoMDFS3bt30+uuv6+mnn9a5c+fUvn17rVu3TjNmzFC7du3UsWNHrVu3zu73\nkJ6ernnz5qlo0aIaOXKkoqOj9dprr0mSSpQooYULFyo2NlYTJkzQsmXLNGrUqJueFyjMCDIUKhcu\nXFBYWJgkKScnR0888YT+/ve/5z1er149SdLevXt17tw5vfHGG5KkjIwMVahQQSdOnFD58uXl4+Mj\nSXrqqad06NChfOeIj4/P66aKFSumOXPm3FDHjh07lJqaqhkzZkiS3NzclJiYqCNHjqhbt26SpKef\nftru91OiRAl169ZNLi4uOnXqlEqVKpX3WMOGDfO+p59++snwvEBhRpChUPn9GtmtuLu7S5I8PDwU\nEBCg2bNn53t83759eZ+tJV0Lwz+y2Ww33X49Dw8PffLJJ/L19c23PTc3Vy4u1y5NZ2dnGx7j7Nmz\nmjBhgmJiYuTn56cJEybcUMcfj3mr8wKFGcMe+FOqU6eO4uPjde7cOUnS6tWrtWHDBlWqVEm//vqr\nLl26pNzcXMXGxt6wb7169bR161ZJUkpKil555RVlZGTIZrMpMzNTkvT4449r9erVkq51iR9++KEk\nqVq1avrhhx8k6abHvl5iYqJ8fHzk5+en5ORkbdu2TRkZGXmPx8XFSbo2LVmjRg3D8wKFGR0Z/pTK\nlCmjYcOGqXv37vLy8pKnp6cmTJig4sWLq0ePHurUqZPKly+v8uXL6+rVq/n2bdWqlfbs2aPQ0FBl\nZ2era9eu8vDwUMOGDfXee+9p6NChGjZsmEaOHKmYmBhlZGSoZ8+ekqTevXtr0KBBWrNmjerVqyc3\nt1v/FaxVq5YqV66sl19+WZUqVVKfPn00atQoNWnSRJKUnJys7t276/Tp03rvvfck6ZbnBQoz3jQY\nAGBpLC0CACyNIAMAWBpBBgCwNIIMAGBpBBkAwNIIMgCApRFkAABLI8gAAJb2/wDO7yh0Mt3k6gAA\nAABJRU5ErkJggg==\n",
            "text/plain": [
              "<matplotlib.figure.Figure at 0x7f48e5208f28>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "iMk7tN1h98x9",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "When we have more than two labels (binary), we have options to calculate the evaluation metrics at a micro/macro level (per clas label), weighted, etc. You can find out more by looking at the [offical docs](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_fscore_support.html)."
      ]
    },
    {
      "metadata": {
        "id": "9v6zc1_1PWnz",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Inference"
      ]
    },
    {
      "metadata": {
        "id": "Zl9euDuMPYTN",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Now let's see if you would've survived the titanic."
      ]
    },
    {
      "metadata": {
        "id": "kX9428-EPUzx",
        "colab_type": "code",
        "outputId": "ef100af7-9861-4900-e9c7-ed6d93c69069",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 80
        }
      },
      "cell_type": "code",
      "source": [
        "# Input your information\n",
        "X_infer = pd.DataFrame([{\"name\": \"Goku Mohandas\", \"cabin\": \"E\", \"ticket\": \"E44\", \n",
        "                         \"pclass\": 1, \"age\": 24, \"sibsp\": 1, \"parch\": 2, \n",
        "                         \"fare\": 100, \"embarked\": \"C\", \"sex\": \"male\"}])\n",
        "X_infer.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>age</th>\n",
              "      <th>cabin</th>\n",
              "      <th>embarked</th>\n",
              "      <th>fare</th>\n",
              "      <th>name</th>\n",
              "      <th>parch</th>\n",
              "      <th>pclass</th>\n",
              "      <th>sex</th>\n",
              "      <th>sibsp</th>\n",
              "      <th>ticket</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>24</td>\n",
              "      <td>E</td>\n",
              "      <td>C</td>\n",
              "      <td>100</td>\n",
              "      <td>Goku Mohandas</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>male</td>\n",
              "      <td>1</td>\n",
              "      <td>E44</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   age cabin embarked  fare           name  parch  pclass   sex  sibsp ticket\n",
              "0   24     E        C   100  Goku Mohandas      2       1  male      1    E44"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 20
        }
      ]
    },
    {
      "metadata": {
        "id": "c6OAAQoaWxAb",
        "colab_type": "code",
        "outputId": "85eb1c6d-6f53-4bd4-bcc3-90d9ebca74c8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 80
        }
      },
      "cell_type": "code",
      "source": [
        "# Apply preprocessing\n",
        "X_infer = preprocess(X_infer)\n",
        "X_infer.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>age</th>\n",
              "      <th>fare</th>\n",
              "      <th>parch</th>\n",
              "      <th>sibsp</th>\n",
              "      <th>pclass_1</th>\n",
              "      <th>embarked_C</th>\n",
              "      <th>sex_male</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>24</td>\n",
              "      <td>100</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   age  fare  parch  sibsp  pclass_1  embarked_C  sex_male\n",
              "0   24   100      2      1         1           1         1"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 21
        }
      ]
    },
    {
      "metadata": {
        "id": "48sj5A0mX5Yw",
        "colab_type": "code",
        "outputId": "d9571238-70ab-427d-f80c-7b13b00efc95",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 80
        }
      },
      "cell_type": "code",
      "source": [
        "# Add missing columns\n",
        "missing_features = set(X_test.columns) - set(X_infer.columns)\n",
        "for feature in missing_features:\n",
        "    X_infer[feature] = 0\n",
        "\n",
        "# Reorganize header\n",
        "X_infer = X_infer[X_train.columns]\n",
        "X_infer.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>age</th>\n",
              "      <th>sibsp</th>\n",
              "      <th>parch</th>\n",
              "      <th>fare</th>\n",
              "      <th>pclass_1</th>\n",
              "      <th>pclass_2</th>\n",
              "      <th>pclass_3</th>\n",
              "      <th>embarked_C</th>\n",
              "      <th>embarked_Q</th>\n",
              "      <th>embarked_S</th>\n",
              "      <th>sex_female</th>\n",
              "      <th>sex_male</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>24</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>100</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   age  sibsp  parch  fare  pclass_1  pclass_2  pclass_3  embarked_C  \\\n",
              "0   24      1      2   100         1         0         0           1   \n",
              "\n",
              "   embarked_Q  embarked_S  sex_female  sex_male  \n",
              "0           0           0           0         1  "
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 22
        }
      ]
    },
    {
      "metadata": {
        "id": "rP_i8w9IXFiM",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Standardize\n",
        "standardized_X_infer = X_scaler.transform(X_infer)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "7O5PbOAvXTzF",
        "colab_type": "code",
        "outputId": "f1c3597e-1676-476f-e970-168e5c3fca6c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "# Predict\n",
        "y_infer = log_reg.predict_proba(standardized_X_infer)\n",
        "classes = {0: \"died\", 1: \"survived\"}\n",
        "_class = np.argmax(y_infer)\n",
        "print (\"Looks like I would've {0} with about {1:.0f}% probability on the Titanic expedition!\".format(\n",
        "    classes[_class], y_infer[0][_class]*100.0))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Looks like I would've survived with about 57% probability on the Titanic expedition!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "8PLPFFP67tvL",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Interpretability"
      ]
    },
    {
      "metadata": {
        "id": "jv6LKNXO7uch",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Which of the features are most influential?"
      ]
    },
    {
      "metadata": {
        "id": "KTSpxbwy7ugl",
        "colab_type": "code",
        "outputId": "b37bf39c-f35d-4793-a479-6e61179fc5e5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "cell_type": "code",
      "source": [
        "# Unstandardize coefficients \n",
        "coef = log_reg.coef_ / X_scaler.scale_\n",
        "intercept = log_reg.intercept_ - np.sum((coef * X_scaler.mean_))\n",
        "print (coef)\n",
        "print (intercept)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[-0.02155712  0.39758992  0.78341184 -0.0070509  -2.71953415  2.01530102\n",
            "   3.50708962  0.11008796  0.         -0.11008796  2.94675085 -2.94675085]]\n",
            "[5.10843738]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "xJgiIupyE0Hd",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "A positive coefficient signifies correlation with the positive class (1=survived) and a negative coefficient signifies correlation with the negative class (0=died)."
      ]
    },
    {
      "metadata": {
        "id": "RKRB0er2C5l-",
        "colab_type": "code",
        "outputId": "39ad0cf3-13b1-4aa8-9a6b-4456b8975a39",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "cell_type": "code",
      "source": [
        "indices = np.argsort(coef)\n",
        "features = list(X_train.columns)\n",
        "print (\"Features correlated with death:\", [features[i] for i in indices[0][:3]])\n",
        "print (\"Features correlated with survival:\", [features[i] for i in indices[0][-3:]])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Features correlated with death: ['sex_male', 'pclass_1', 'embarked_S']\n",
            "Features correlated with survival: ['pclass_2', 'sex_female', 'pclass_3']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "RhhFw3Kg-4aL",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Proof for unstandardizing coefficients:\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "ER0HFHXj-4h8",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Note that only X was standardized.\n",
        "\n",
        "$\\mathbb{E}[y] = W_0 + \\sum_{j=1}^{k}W_jz_j$\n",
        "\n",
        "$z_j = \\frac{x_j - \\bar{x}_j}{\\sigma_j}$\n",
        "\n",
        "$ \\hat{y} = \\hat{W_0} + \\sum_{j=1}^{k}\\hat{W_j}z_j $\n",
        "\n",
        "$\\hat{y} = \\hat{W_0} + \\sum_{j=1}^{k} \\hat{W}_j (\\frac{x_j - \\bar{x}_j}{\\sigma_j}) $\n",
        "\n",
        "$\\hat{y} = (\\hat{W_0} - \\sum_{j=1}^{k} \\hat{W}_j\\frac{\\bar{x}_j}{\\sigma_j}) +  \\sum_{j=1}^{k} (\\frac{\\hat{w}_j}{\\sigma_j})x_j$"
      ]
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "5yBZLVHwGKSj"
      },
      "cell_type": "markdown",
      "source": [
        "# K-fold cross validation"
      ]
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "fHyLTMAAGJ_x"
      },
      "cell_type": "markdown",
      "source": [
        "Cross validation is a resampling techniques to evaluate a model. Instead of splitting out data once at the beginning into train/val/test sets. We do this k (usually k=5 or 10) times with different training and evaluation sets.\n",
        "\n",
        "Steps:\n",
        "1.   Shuffle the *train* dataset randomly.\n",
        "2.   Split the dataset into k discint groups.\n",
        "3.   For each iteration k, choose one of the groups to be your test set and the rest as your training set.\n",
        "4.   Repeat so that each group experiences being part of the test and train set.\n",
        "5.   Train a model using randomly initialzied weights.\n",
        "6.   After each iteration k, reinitialize the model with the same randomly initialzied weights and repeat on the new test set.\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "6XB6X1b0KcvJ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from sklearn.model_selection import cross_val_score"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "UIqKmAEtVWMg",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# K-fold cross validation\n",
        "log_reg = SGDClassifier(loss=\"log\", penalty=\"none\", max_iter=args.num_epochs)\n",
        "scores = cross_val_score(log_reg, standardized_X_train, y_train, cv=10, scoring=\"accuracy\")\n",
        "print(\"Scores:\", scores)\n",
        "print(\"Mean:\", scores.mean())\n",
        "print(\"Standard Deviation:\", scores.std())"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "L0aQUomQoni1",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# TODO"
      ]
    },
    {
      "metadata": {
        "id": "jCpKSu53EA9-",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "- interaction terms\n",
        "- interpreting odds ratio\n",
        "- simple example with coordinate descent method (sklearn.linear_model.LogisticRegression)"
      ]
    }
  ]
}